Method and system for virtual intelligence user interaction

ABSTRACT

A method and apparatus to generate and update virtual personification using artificial intelligence comprising a system configured to perform the following. Receive data associated with a person such as text files, audio files, image files, and video files. Render a virtual personification of the person and output the virtual personification to a user, such as on a display screen. Then, receiving and interpreting a user input to generate a user request, and then updating the virtual personification. The update may include generating an audio output using the text files and the audio files of the person and/or generating a video output using the image files and the video files of the person. The audio output and the video output is presented to the user by the virtual personification and it has not previously occurred by the person or thing represented by the virtual personification.

1. FIELD OF THE INVENTION

The present invention is directed to a method and system to provide userinteraction with virtual personifications using artificial intelligence(“AI”).

2. DESCRIPTION OF THE RELATED ART

Advancement with VR and AR technology now allows for users to view realor simulated environments, (referred to as virtual environments) using ascreen equipped headset or a traditional screen. Within these virtualenvironments, user have been able to view elements and move about tofurther explore the world. However, user interaction with currenttechnology on virtual avatar is typically based on pre-recorded,pre-scripted, image or audio files. In other words, the user can lookabout the environment and travel from place to place within theenvironment but beyond that, interaction with the virtual environment islimited.

Other systems allow for some interaction with the virtual environment toobtain information about an items in the environment, such as to clickon an items to obtain additional information. However, the interactionwith elements in the virtual environment is limited to pre-created orpre-recorded information that typically is no more than a shortpre-recorded message or text that is typically non-responsive and oftenno better than a frustrating voice script. These systems lackindividualization to the particular user's interest and specificquestions and are sterile in that prior art systems are no better thansimply reading an article or watching a video on a web site.

SUMMARY

To overcome the drawbacks of the prior art and provide additionalbenefits, disclosed is a system and method to generate and updatevirtual personification using artificial intelligence receiving dataassociated with a person, the data comprising one or more of thefollowing: text files, audio files, image files, and video files, andrendering a virtual personification of the person and outputting thevirtual personification to a user. Then, receiving and interpreting auser input to generate a user request and updating the virtualpersonification in response to the user request. The update comprisingone or more of the following. Responsive to the user request, generatingan audio output using the text and audio files of the person andresponsive to the user request, generating a video output using theimage files and the video files of the person, such that the audiooutput and the video output is presented to the user by the virtualpersonification. Furthermore, the audio output and the video outputpresented by the virtual personification has not previously occurred bythe person or thing represented by the virtual personification.

In one embodiment, the virtual personification is of a person, eitherliving or deceased. It is contemplated that the virtual personificationmay comprise an audio output and video output which are presented in avirtual environment of a type associated with the virtualpersonification. The virtual personification may comprise arepresentation of a non-living item.

In one embodiment, the method is further configured to, responsive tobeing unable to create the generated response at the virtual realitydevice, transmit the question or request from the user to a remoteartificial intelligence module. The remote artificial intelligencemodule may be a computing device with a processor and memory storingmachine readable code configured to receive the question or request fromthe user via the virtual reality device, process the question or requestto derive a meaning, and perform one or more searches for answers to thequestion or request in databases unrelated to the virtualpersonification. Upon locating an answer to the question or request,generating data that represents the virtual personification answeringthe question or request and transmitting the answer or the data thatrepresents the virtual personification answering the question or requestto the virtual reality device for presentation to the user. It is alsocontemplated that the method may further comprise tracking a user's handposition using one or more user hand position tracking devices todetermine what the user is pointing at in the virtual environment. Thestep of generating a response to the question or request may useartificial intelligence to generate an answer by searching one or moredatabases that contain information from the person represented by thevirtual personification but which to not provide a direct answer to thequestion or request.

Also disclosed is a system for presenting an interactive, artificialintelligence assisted, virtual personification to a user comprising avirtual reality device configured to have at least a portion be worn bythe user. The virtual reality device includes a wearable screenconfigured for viewing by a user, one or more speakers configured toprovide audio output to the user, a microphone configured to receiveaudio input from the user, and one or more external input devicesconfigured to receive input from the user. Also part of the virtualreality device includes a communication module configured to communicateover a computer network or Internet, and a processor with access to amemory. The processor executes machine readable code and the memory isconfigured to store the machine readable code. The machine readable codeis configured to present a virtual environment on the wearable screenand through the one or more speakers to the user and present, to theuser on the wearable screen and through the one or more speakers, avirtual personification of a person currently living or deceased, in thevirtual environment. The code is also configured to receive a questionor request from the user regarding one or more aspects of the virtualenvironment or the virtual personification and then generate a responseto the question or request from the user, which includes generatingvideo content and audio content which did not previously exist. The codethen presents the generated response to the user on the wearable screenand through the one or more speakers in response to question or requestfrom the user.

In one embodiment, the machine readable code is further configured to,responsive to being unable to create the generated response at thevirtual reality device, transmit the question or request from the userto a remote artificial intelligence module. It is further contemplatedthat the remote artificial intelligence module may be a computing devicewith memory and processor such that memory store machine readable codeconfigured to receive the question or request from the user via thevirtual reality device, process the question or request to derive ameaning, and perform one or more searches for answers to the question orrequest in databases unrelated to the virtual personification. Then,upon locating an answer to the question or request, generating data thatrepresents the virtual personification answering the question orrequest, and transmitting the answer or the data that represents thevirtual personification answering the question or request to the virtualreality device for presentation to the user.

The system may further comprise one or more user hand position trackingdevice configured to track a position of a user's hand to determine whatthe user is pointing at in the virtual environment. In one embodiment,the input from the user comprises an audio input or an input to the oneor more external input devices. It is contemplated that generating videocontent and audio content which did not previously exist is generated byprocessing existing video, audio, or both, of the person represented bythe virtual personification, to form the video content and audio contentwhich did not previously exist. In addition, the generated response tothe question or request uses artificial intelligence to generate ananswer by searching one or more databases that contain information froma person represented by the virtual personification but which to notprovide a direct answer to the question or request.

Also disclosed herein is a method for presenting an interactiveexperience with a virtual personification using a screen, speakers, andmicrophone of a user computing device. In one embodiment the methodcomprises a virtual environment on the wearable screen and through theone or more speakers to the user and present the virtual personificationin the virtual environment. Then, receiving input from the usercomprising a question, a user request, or subject regarding one or moreaspects of the virtual environment, the virtual personification, or theactions of the virtual personification in the virtual environment. Thismethod then sends a request for a response to the input from the user toan AI computing device that is remote from the user computing device,and with the AI computing device, create a response based onpre-existing content stored in one or more databases which is processedto create the generated response. Then, transmitting the generatedresponse to the user computing device and, at the user computing device,based on the generated response from the AI computing device, generatingvideo content and audio content which did not previously exist. Finally,the method of operation presents video content and audio content whichdid not previously exist to the user.

In one embodiment, the AI computing device is a computing device withmemory and processor such that the memory stores machine readable codeconfigured to receive the input from the user computing device, processthe input from the user to derive a meaning, and based on the meaning,perform one or more searches for answers to the input from the user indatabases unrelated to the virtual personification. Upon locating anresponse to the input from the user, generate data that represents thevirtual personification answering the question or request, and transmitthe data, that represents the virtual personification responding to theinput from the user, to the user computing device.

This method may further include monitoring one or more user handposition tracking devices configured to track a position of a user'shand to determine what the user is pointing at in the virtualenvironment and interpreting the pointing as the input from the user. Itis contemplated that the input from the user comprises an audio input oran input from the user to the one or more external input devices. Thestep of generating video content and audio content which did notpreviously exist occurs by processing existing video, audio, or both ofa person represented by the virtual personification to generate newcontent.

Other systems, methods, features and advantages of the invention will beor will become apparent to one with skill in the art upon examination ofthe following figures and detailed description. It is intended that allsuch additional systems, methods, features and advantages be includedwithin this description, be within the scope of the invention, and beprotected by the accompanying claims.

DESCRIPTION OF THE FIGURES

The components in the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of the invention.In the figures, like reference numerals designate corresponding partsthroughout the different views.

FIG. 1A illustrates a first exemplary embodiment of the present virtualpersonification AI system integrated into a virtual reality system.

FIG. 1B illustrates a second exemplary embodiment of the virtualpersonification AI system which may use a local AI operating on aseparate user device such as a smartphone, a tablet, a personalcomputer, etc.

FIG. 2 illustrates an exemplary environment of use of the virtualpersonification AI system.

FIG. 3 illustrates a block diagram of an example embodiment of acomputing device, also referred to as a user device which may or may notbe mobile.

FIG. 4 is a block diagram of an exemplary computing device, mobiledevice, or server, such as one of the devices described above, accordingto one exemplary embodiment.

DETAILED DESCRIPTION OF THE INVENTION Glossary of Terms

AI services: Services provided as procedures and methods to a program toaccomplish artificial intelligence goals. Examples may include, but arenot limited to, image modeling, text modeling, forecasting, planning,recommendations, search, speech processing, audio processing, audiogeneration, text generation, image generation, and many more.

Device: A device is any element running with a minimum of a CPU or asystem which is used to interface with a device. Optionally, anaccelerator can be attached in the form of a GPU or other specializedhardware accelerator. This accelerator can speed up the computation ofAI services.

Application: An application is any software running on any device suchas mobile devices, laptop, desktop, server, smart watches, tablets, homespeakers, wearable devices including smart rings, glasses, hearing aids,CarPlay devices, security cameras, webcams, televisions, projectionscreen monitors, sound bars, personal computers, headphones, earbuds,and laptop devices where a user can interact with touch, audio, visual,or passively.

The following terms are used in this document and the followingdefinitions are provided to aid in understanding but should beinterpreted as being limiting in scope

AI services: Services provided as procedures and methods to a program toaccomplish artificial intelligence goals. Examples may include, but arenot limited to, image modeling, text modeling, forecasting, planning,recommendations, search, speech processing, audio processing, audiogeneration, text generation, image generation, and many more.

Device: A device is any element running with a minimum of a CPU or asystem which is used to interface with a device. Optionally, anaccelerator can be attached in the form of a GPU or other specializedhardware accelerator. This accelerator can speed up the computation ofAI services.

Application: An application is any software running on any device suchas mobile devices, laptop, desktop, server, smart watches, tablets, homespeakers, wearable devices including smart rings, glasses, hearing aids,CarPlay devices, security cameras, webcams, televisions, projectionscreen monitors, sound bars, personal computers, headphones, earbuds,and laptop devices where a user can interact with touch, audio, visual,or passively.

In this disclosure, a virtual personification system may analyzepre-recorded data to generate dynamic responses to userrequests/questions through virtual personifications. In one embodiment,the virtual personification may be a virtual representation which may bebased on a real person. For example, the user, a family member orrelative, a famous person, a historical figure, or any other typeperson. The virtual representation may also be a user or computercreated person that does not represent a real person. Pre-recorded datamay include image, video, or audio footage of the real person (such asYouTube and other film footage). Dynamic responses are generated to userrequests/questions related to that known person, even thoughpre-recorded data may not include any adequate responses or responseswhich will match the question.

For example, a user may wish to be provided with a recipe from a famouschef, such as Gordon Ramsey, to make grilled salmon. Upon determinationthat pre-recorded data exists on Gordon Ramsey making another type ofgrilled chicken, the virtual personification may analyze Gordon Ramsey'sfootage on making grilled chicken and grilled potatoes to generate avirtual personification of Gordon Ramsey guiding the user through theprocess of making grilled salmon, as if Gordon Ramsey were in a cookingshow and personally providing detailed instructions to the specific userrequest. The system AI can pull details from prior recordings andmanipulate the visual and audio files to create a new virtualrepresentation that is directly and accurately responsive to the user'srequest. AI may generate new information, such as how to adjust theresponse to be responsive to the specific user request. In the exampleof the cooking question, AI can understand the user's request, analyzethe information already provided by the chef about how to cook chicken,realize that chicken is not salmon, and then search for a recipe forsalmon by the same chef or recipe, and then process the new recipe andthe virtual representation to present the new recipe to the user of thesystem using the virtual representation, as if the original chef wasactually providing the recipe for salmon and not chicken. Althoughapplied to food, this example may be applied to any other topic orenvironment of use.

The virtual personification of Gordon Ramsey may use a voice that soundslike Gordon Ramsey, may be dressed like Gordon Ramsey, as he typicallyappears on cooking shows, and may mimic Gordon Ramsey's body languageand speech pattern. AI may be used to create the virtual personificationeven in situations when the actual person never actually provided aresponsive answer in a video or audio recording. The virtualrepresentation may be created using built-in AI modules such as avirtual personification rendering module (discussed in more detailsbelow) or using third-party tools, which the virtual personificationsystem may interface with.

In another example, the user may attempt Gordon Ramsey's recipe ofscrambled eggs, which may already be available on YouTube, and which mayinvolve the use of milk. However, upon determination he has no milk inthe fridge, the user may wish to ask Gordon Ramsey whether whipped creammay be used as a substitute. While in the existing footage on YouTube,Gordon Ramsey may not have provided an answer to that question, thevirtual personification may analyze Gordon Ramsey's footage onsubstituting other items for milk to generate a virtual personificationof Gordon Ramsey to answer this user question. The virtualpersonification of Gordon Ramsey may include a prediction of GordonRamsey's typical reaction in such situations. For example, the AI maydetermine, based on pre-recorded data, that Gordon Ramsey typically actsimpatiently to such questions. Thus, the virtual personification ofGordon Ramsey may display a frown or curt gestures when providing thepredicted answer.

In one embodiment, the virtual personification may be presented in avirtual reality space, which may be rendered using a virtual realitysystem. For example, in a cooking environment, the virtual reality spacemay be a kitchen. For other topics, such as carpentry the environmentmay be a wood working shop, car repair would appear in an auto garage,education may appear as a classroom, and information about a topic mayactually appear inside the items, such as inside a virtual computer or avirtual engine to show how something works in combination with AI thatcreates answers for the user using the virtual reality space and thevirtual personification.

FIG. 1A illustrates a first exemplary embodiment of the present virtualpersonification AI system integrated into a virtual reality system. Thevirtual reality space is rendered by a virtual reality system. Exemplaryvirtual reality systems are described in U.S. Pat. No. 9,898,091, U.S.Patent Publication 2014/0364212, and U.S. Patent Publication2015/0234189, which are incorporated by reference herein in theirentirety as teaching exemplary virtual reality systems and methods. Auser 100A may access the virtual reality space by the one or morecomponents of a virtual reality system, such as a virtual reality device(“VR device”) 104A and external input devices 108A, which may beaccessories to the VR device 104A. The VR device 104A may be in directcommunication with the external input devices 108A (such as byBluetooth®) or via network 112A providing internet or signals (e.g., apersonal area network, a local area network (“LAN”), a wireless LAN, awide area network, etc.). The VR device 104A may also communicate with aremote AI 116A via the network 112A.

In a preferred embodiment, the VR device 104A may be a wearable userdevice such as a virtual reality headset (“VR headset”), and theexternal input devices 108A may be hand-held controllers where a usermay provide additional input such as arm motion, hand gestures, andvarious selection or control input through buttons or joysticks on suchcontrollers.

The VR device may generally include input devices 120A through 128A,input processing modules 132A, VR applications 134A, output renderingmodules 138A, output devices 156A, 160A, and a communication module164A. Input devices may include one or more audio input devices 120A(such as microphones), one or more position tracking input devices 124A(to detect a user's position and motion), and one or more facialtracking input devices 128A (such as facial cameras to detect facialexpressions, eye-tracking camera to detect gaze and eye movement, etc.).Additional external input devices may provide user biometrics data ortracking of other user body parts.

The input processing modules 132A may include, but are not limited to,an external input processing module 142A (used to process externalinputs such as input from external devices 108A or additional externalinput devices discussed above), an audio input processing module 144A(used to process audio inputs, such as user speech or sounds), aposition input processing module 146A (to process position and motiontracking inputs such as hand motions, finger motions, arm motions, headposition), and a facial input processing module 148A (to process facialinputs of the user).

The VR applications 134A are generally responsible for rendering virtualreality spaces associated with their respective VR applications 134A.For example, a VR museum application may render a virtual museum throughwhich a user may traverse and present various artwork which the user mayview or interact with. This is achieved through the VR application's134A integration with output rendering modules 138A, which in turnpresents the rendered files on output devices 156A, 160A.

Specifically, the output rendering modules 138A may include, but are notlimited to, an audio output processing module 150A responsible forprocessing audio files, and an image and/or video output processingmodule 152A, responsible for processing image and/or video files. Inturn, one or more audio output devices 156A, such as built-in speakerson the VR headset may present the processed audio file, and one or moreimage and/or video output devices 160A (such as a built-in screen on theVR headset) may display the processed image and/or video files. Othertypes of output may include, but are not limited to, motion ortemperature changes to the VR device 104A or the external input devices108A (such as vibration on hand-held controllers).

User interaction may in turn modify the virtual reality space. Forexample, if a user inputs motion to indicate he picked up a vase, therendered virtual reality space may display a vase moving in accordancewith the user's motion. Thus, the transmission of information occurs ina bi-directional streaming fashion, from the user 100A to the VR device104A and/or external input devices 108A, then from the VR device 104Aand/or external input devices 108A back to the user 100A. U.S.application Ser. No. 17/218,021 provides a more detailed discussion onbi-directional streaming using AI services and examples of broader andspecific uses.

The AI may be completely or partially built into the VR device 104A orspecific VR applications 134A. Such built-in AI components may bereferred to a local AI 168A. Other AI components may be located in theremote AI 116A, which may be operating on remote devices or oncloud-based servers. The local and remote AI 168A, 116A may communicatevia the network 112A.

The AI may enhance the user's 100A interaction with the virtual realitysystem using the embodiments and methods described above. The AI mayinclude one or more of the following components to generally operate theAI and process data, one or more processors 172 and one or more memorystorage devices where logic modules 176 and machine learning modules 178may be stored to provide general AI services. The memory storage devicesmay further include one or more modules to specifically enhance user-VRinteraction, such as speech-to-text modules 180, non-verbal inputprocessing modules 182, text augmentation modules 184, conversationmanagement modules 186, response generation modules 188, audio renderingand updating modules 190, virtual personification rendering modules 192,virtual personification prediction modules 194, and integration modules196.

The speech-to-text modules 180 may be used to perform voice detectionand customized speech to text recognition, as well as to generallydetect, recognize, process, and interpret user audio input. Recognitionallows the speech-to-text modules 180 to distinguish between verbalinput (such as a user question) and non-verbal input (such as the user'ssigh of relief).

A user may start an active conversation in the virtual reality space bysimply speaking. The speech-to-text modules 180 may use voice activitydetection in order to differentiate that the user has started speaking,as opposed to ambient noise activity. When true speech is detected, thespeech-to-text modules 180 may process the input audio from themicrophone to recognize the user's spoken text. This processing caneither happen as part of the viewing device (such as the VR device104A), on a device connected to the viewing device, or on a remoteserver over the network (such as the remote AI 116A). This process mayconvert the stream of audio into the spoken language, such as textprocessable by a computer.

The speech-to-text modules 180 may be customized to the current scenethat the user is experiencing inside the virtual space, or a virtualpersonification that the user wishes to interact with. Thiscustomization could allow for custom vocabulary to be recognized when itwould make sense in the specific environment or specific virtualpersonification. For example, if a user were interacting with a virtualpersonification of a cooking chef, then the speech recognition systemmay be customized to enhance name recognition for words associated food,whereas in a different environment a different vocabulary would be used.If the virtual personification of Gordon Ramsey were in a kitchen, thenthe speech recognition system may be customized to enhance namerecognition for kitchen utensils.

While the virtual reality system may have its own modules to processaudio inputs, the AI's speech-to-text modules 180 are intended tointegrate and enhance existing features in the virtual reality system.For example, the AI speech-to-text modules 180 may generate exponentialamounts of interpretations from a single user input, automaticallyselect the top interpretation based on user data, and hold multi-turnconversations with the user as a continuation of that single user input.Appendix A includes a more detailed discussion on systems and methodsfor enhanced speech-to-text. The enhanced speech-to-text and integrationwith other applications outside the virtual reality system, and theadditional mechanism to recognize usable user input (as discussed instep 2) and to process out of scope user input.

The AI's non-verbal input processing modules 182 may be used to processnon-verbal input. As discussed above, audio input may be non-verbal(such as a user's sigh of relief, or tone of voice). As well, externalinput devices 108A may include devices to track a user's biometrics orbody parts other than arm, hand, and finger movement. Examples ofdevices to track a user's biometrics include but are not limited tosmartwatches, Fitbits™, heart-rate monitors, blood pressure monitors, orany other devices which may be used to track a user's heart-rate, oxygenlevel, blood pressure, or any other metrics that may track a user's bodycondition. Such input may all be processed using additional processingmodules, which may be part of the virtual reality system (such as builtinto the VR device 104A), and/or may be part of the local or remote AI168A, 116A.

The text augmentation modules 184 may be used to add further context tothe interpreted user 100A input. When the speech-to-text modules 180 hassuccessfully transcribed the user's spoken text, the text augmentationmodules 184 may supplement the spoken text with what the user iscurrently doing, or interacting with, to enhance its linguisticunderstanding of what the user has said. For example, this allows the AIto find co-references between what the user said and what they arelooking at. Such as, if a user asks, “how old is this”, the term “this”can be implied from what the user is currently looking at, touching,near, or pointing at in the virtual world. This functionality can becarried about by fusions of any or one of the following inputs: theuser's head position, eye detection, hand position—including placement,grip, pointing, controller position, and general orientation.Furthermore, the system may also fuse in non-controller related signals,such as biometrics from heart rate, breathing patterns, and any otherbio sensory information. This information is fused over time to detectnot just instantaneous values for fusion but trends as well.

The text augmentation modules 184 may also be integrated with thenon-verbal input processing modules 182 to receive further context. Forexample, in a multi-turn conversation where a user requests information,the user may input the word “okay”. Conventional system may, by default,cease communication because the response “okay” may be pre-coded as acommand to terminate interaction. The text augmentation modules 184, incontrast, may analyze the user's tone to detect (1) boredom, andinterpret “okay” as a request to shorten the information provided, (2)hesitation or confusion, and interpret “okay” as a request foradditional information, (3) impatience, and interpret “okay” as arequest to end the interaction.

The text augmentation modules' 184 integration with other devices andmodules may not be linear. Rather, context from the virtual realitysystem may be used in one or more steps of speech interpretation. Forexample, in a multi-turn conversation (such as the conversationdescribed above), at each turn of a user input, the speech-to-textmodules may be used to generate the most accurate interpretation of theuser's input, and the non-verbal input processing module 182 may be usedto inject more context. Further, the AI's conversation managementmodules 186 may be integrated with the text augmentation modules 184 togenerate the output used in single or multi-turn conversations.

Once the AI has augmented the text by considering the current state ofthe virtual space in relation to the user, then a conversation may becarried out. The conversation management modules 186 may classify thespoken text into different categories to facilitate the open-endedconversation. First the conversation management modules 186 maydetermine if a statement is meant to initiate a new conversation or onethat continues an existing conversation. If the user is detected toinitiate a new conversation, then the conversation management modules186 may classify the result among categories. A first category mayinclude user comments that may not necessarily require a strongresponse. For example, if a user states “this is really cool”, theconversation management modules 186 may render the virtualpersonification to respond with a more descriptive or expressiveresponse in relation to what was remarked as being cool. Alternatively,the virtual personification may not respond. A second category mayinclude user questions that may be in relation to the current scene. Athird category may be user questions that are in relation to thenon-virtualized world (i.e., reality).

In the second and third categories, the conversation management modules186 may facilitate an answer to the question via the virtualpersonification. In the second category of a question being detected inrelation to the virtual world, the system may then proceed down to oneof two or more paths. The conversation management modules 186 may firstattempt to use information in pre-recorded data to answer the question.For example, during a user interaction with a virtual Gordon Ramsey onmaking a grilled salmon, a user may ask about the use of an ingredientnot in the current recipe. The conversation management modules 186 mayretrieve footage from another video where Gordon Ramsey uses thatingredient and may render the virtual Gordon Ramsey to modify thecurrent recipe to include that ingredient.

If no pre-recorded data exists, then the conversation management modules186 request the response generation modules 188 to analyze additionaldata (such as data on Gordon Ramsey's presentation of a similaralternative ingredient or based on other chefs or known cookinginformation) to generate new behavior, speak, actions, response for thevirtual Gordon Ramsey (such as the output of an opinion that theingredient may not be desirable, or the rendering of Gordon Ramseyadding the ingredient to the recipe using Gordon Ramsey's voice andbehavior). If the user is in an existing conversation, then theconversation management modules 186 may proceed with the same approachas in the previous section, but with the added impetus of consideringthe context. Using context and past conversation details in the AIsystem provides a more realistic user interaction and avoid the virtualrepresentation from repeating themselves or providing the same response.

After the response is provided, the conversation management modules 186may account for the likelihood that the user will continue to askquestions or follow ups to the previous response. The conversationmanagement modules 186 may use this information to better carry out thenext algorithm by utilizing this additional information.

The audio rendering and updating modules 190 may be used, bothindependently and/or in conjunction with other modules 188, to outputaudio. The audio rendering and updating modules 190 may ensure, forexample, that the proper response to a user question to virtual GordonRamsey may indeed be in Gordon Ramsey's voice, tone, and speech pattern.For example, based on analysis of Gordon Ramsey's past audio files, theaudio rendering and updating modules 190 may determine Gordon Ramseyfrequently accompanies his speech with an exasperated sigh. As a result,the audio rendering and updating modules 190 may cause the virtualpersonification of Gordon Ramsey to output the same exasperated sigh inits user interactions. Additional analysis may be performed to customizespeech of the virtual personification in the following areas: volume(such as a person who frequently speaks in raised voice), accent, tone(such as a person who likes to emphasize certain words), speech pattern,etc.

The audio rendering and updating modules 190 may also supplement theuser experience with the appropriate background noise or music, whichmay be updated based on user action. For example, as virtual GordonRamsey guides the user through the experience of baking a cake, atone-point virtual Gordon Ramsey may show the user how to turn on a cakemixer. In addition to the visual rendering of a cake mixer that iscurrently in use, the audio rendering and updating modules 190 mayupdate the ambient noise to a background noise of an operating cakemixer. At a later point, then virtual Gordon Ramsey turns the cake mixeroff, the audio rendering and updating modules 190 may update thebackground noise of an operating cake mixer back to ambient noise. Thiswould be true for any and all sounds in a virtual environment.

The virtual personification rendering module 192 may be used, inconjunction with other modules, to output the visual representation ofthe virtual personification. For example, the virtual personificationrendering module 192 may be used to generate the virtual Gordon Ramsey,which may include a visual representation of one or more of thefollowing: Gordon Ramsey's face, body, and clothing. The virtualpersonification rendering module 192 processes existing image or videoof the person or item being personified and maps and creates new videofrom existing video and is also able to create new video or image fromprior image and video to create a virtual personification that is doingor saying things which the original person never did or said.

The virtual personification prediction modules 194 may be used, inconjunction with other modules, to predict behavior and responses by thevirtual personification. Particularly in cases where no existingpre-recording data exists, and the AI must generate newresponses/behavior, the virtual personification rendering module 192 maybe used to render predictions on the virtual personifications'responses, which may include one or more of the following: facialexpressions, body movements, gestures, etc.

The integration modules 196 may be used to integrate the one or moremodules described above, so they may operate in sync. For example, theresponse generation modules 188 may be integrated with the audiorendering and updating module to accurately output a virtualpersonification's speech using the proper voice and tone, which may, inturn, be integrated with the virtual personification rendering modules192 and the virtual personification prediction modules 194 to ensure thecoherence between the output of voice, facial expression, and bodymovements. In one embodiment, additional customization may be added tothe virtual personification through additional modules. For example,additional modules may be added to analyze a person's temperament orpersonality and incorporate them into the virtual personification ofthat person.

Conventional virtual reality systems, and conversational AI systems, maynot have the ability to hold multi-turn conversations. For example,following the user's request to add a first ingredient to a recipe,Gordon Ramsey may suggest the use of a second ingredient instead. Theuser may ask, “can you show me what that would look like”. Conventionalvirtual reality systems may not understand what “that” is referring to,or incorrectly identify the user is still looking for a display of thefirst ingredient, or not have the requested information. In addition,the AI system would generate an image or video with what that addedingredient would look like even if such video or image footage neverpreviously existed. In contrast, through the integration ofspeech-to-text modules 180, conversation management modules 186, andlogic modules 176 such as natural language understanding modules, fuzzylogic modules, the AI would understand the user 100A, in continuation ofthe previous exchange, is referring to the second ingredient and maythus provide output that is truly responsive to the user's second input.

In addition to the robust speech interpretation and multi-conversationfeature, the AI presents two other main improvements on conventionalsystems—unlimited data and analysis, and dynamic and predictiverendering. Using the previous example, a conventional rendering of avirtual Gordon Ramsey may be limited to existing video footage of GordonRamsey (such as, for example, a deep fake using Gordon Ramsey's face, orthe output of a pre-recording of Gordon Ramsey). Further, a conventionalsoftware application used to render the virtual Gordon Ramsey may belimited to existing data in the local device. The AI, in contrast, mayquery any database accessible over the network to retrieve additionalpre-recorded data, or generate rendering of virtual Gordon Ramsey innever-before-seen situations based on analysis of existing footage.Thus, not only is the entire universe of information available on theinternet accessible by the user through AI, but additional informationnot currently existing on the internet may be predicted and generated.

FIG. 1B illustrates a second exemplary embodiment of the virtualpersonification AI system which may use a local AI 168B operating on aseparate user device 190 such as a smartphone, a tablet, a personalcomputer, etc. Such separate user devices 190 may establish directcommunication to the communication module 164 in the VR device 104Band/or a remote AI 116B housing additional AI modules or may communicatewith the VR device 104B and remote AI 116B via the network 112B. Thevarious AI components 172-196 illustrated in FIG. 1A and discussed abovemay be stored in the local AI 168B and/or the remote AI 116B. Theexternal input devices 108B and the various VR device components120B-164B may interact with each other and with the local and remote AIs168B, 116B in similar fashion as described in FIG. 1A.

It is contemplated that in any embodiment, including but not limited to1A and 1B, any one or more of the AI modules 172-196 may be included inthe local AI 168 and/or the remote AI 116. In one embodiment, all AImodules 172-196 may be located on a local AI 168 operating in the VRdevice 104 such that no remote AI 116 may be necessary. Alternatively,all AI modules 172-196 may be located in a remote AI 116. In preferredembodiments, most or all AI modules 172-196 are in a remote AI 116 suchthat the AI may be integrated with any VR device 104, including VRdevices 104 with no built-in local AI 168A. Such integration may beachieved using AI layers to power cross platform AI services, which isdiscussed in more details in U.S. application Ser. No. 17/218,021.

While VR devices may be the preferred device to implement the virtualpersonification AI system, it is intended that the virtualpersonification AI system may be used on any computing devices capableof performing user interaction. For example, the virtual personificationAI system may be implemented on a device capable of performing ARdisplay, such that the virtual personification may be output via ARtechnology. Similarly, the virtual personification AI system may beimplemented on smartphones, tablets, personal computers, laptop devices,etc., where the virtual personification may be a 2-dimensional output ona display screen. In one embodiment, the virtual personification may bein audio-only mode, for implementation on a peripheral device (such as avehicle with CarPlay) or wearable devices (such as smartwatches, smartrings, glasses, hearing aids, headphones, earbuds, etc.), home devices(such as home speakers, security cameras, webcams, televisions,projection screen monitors, sound bars, etc.), or any other electronicdevices.

FIG. 2 illustrates an exemplary environment of use of the virtualpersonification AI system. In FIG. 2 , a user 200 may interact with thefollowing devices, which may, as discussed above, be capable ofimplementing the virtual personification AI system: a VR device 204 (asillustrated in FIG. 1 ), an AR device 208, or any other computing device212 (such as a computer, a smart TV, or a smartphone). These devices204-212 may then implement the AI modules 220 (which are separatelyillustrated as 172-196 in FIG. 1A and discussed above). Suchimplementation may be, as discussed above, locally, remotely, or both.

In one embodiment, the AI modules 220 may be integrated with third-partytools such as virtual representation modules 216 and audio data modules224. Virtual representation modules 216 may be any additional tools usedto generate virtual personifications and virtual environments. Audiodata modules 224 may be additional tools used to generate audio forvirtual personifications and virtual environments.

The AI modules 220 and its integrated third-party tools 216, 224 may bein direct communication, or communicate via a network 228 to accessprograms, servers, and/or databases stored in a cloud 232 and/orcloud-based servers, as well as other devices 236, which may in turn beconnected to their respective databases 240. In one embodiment, the AImodules 220 may access the third-party tools 216, 224 remotely, such asvia the network 228. The AI modules 220 may thus access resources fromall connected programs, devices, servers, and/or databases.

FIG. 3 illustrates an example embodiment of a mobile device on which asolution generator may operate, also referred to as a user device whichmay or may not be mobile. This is but one possible mobile deviceconfiguration and as such, it is contemplated that one of ordinary skillin the art may differently configure the mobile device. The mobiledevice 300 may comprise any type of mobile communication device capableof performing as described below. The mobile device may comprise aPersonal Digital Assistant (“PDA”), cellular telephone, smart phone,tablet PC, wireless electronic pad, an IoT device, a “wearable”electronic device or any other computing device.

In this example embodiment, the mobile device 300 is configured with anouter housing 304 designed to protect and contain the componentsdescribed below. Within the housing 304 is a processor 308 and a firstand second bus 312A, 312B (collectively 312). The processor 308communicates over the buses 312 with the other components of the mobiledevice 300. The processor 308 may comprise any type processor orcontroller capable of performing as described herein. The processor 308may comprise a general purpose processor, ASIC, ARM, DSP, controller, orany other type processing device. The processor 308 and other elementsof the mobile device 300 receive power from a battery 320 or other powersource. An electrical interface 324 provides one or more electricalports to electrically interface with the mobile device, such as with asecond electronic device, computer, a medical device, or a powersupply/charging device. The interface 324 may comprise any typeelectrical interface or connector format.

One or more memories 310 are part of the mobile device 300 for storageof machine readable code for execution on the processor 308 and forstorage of data, such as image, audio, user, location, accelerometer, orany other type of data. The memory 310 may comprise RAM, ROM, flashmemory, optical memory, or micro-drive memory. The machine readable code(software modules and/or routines) as described herein isnon-transitory.

As part of this embodiment, the processor 308 connects to a userinterface 316. The user interface 316 may comprise any system or deviceconfigured to accept user input to control the mobile device. The userinterface 316 may comprise one or more of the following: microphone,keyboard, roller ball, buttons, wheels, pointer key, touch pad, andtouch screen. Also provide is a touch screen controller 330 whichinterfaces through the bus 312 and connects to a display 328.

The display comprises any type display screen configured to displayvisual information to the user. The screen may comprise a LED, LCD, thinfilm transistor screen, OEL CSTN (color super twisted nematic), TFT(thin film transistor), TFD (thin film diode), OLED (organiclight-emitting diode), AMOLED display (active-matrix organiclight-emitting diode), capacitive touch screen, resistive touch screenor any combination of such technologies. The display 328 receivessignals from the processor 308, and these signals are translated by thedisplay into text and images as is understood in the art. The display328 may further comprise a display processor (not shown) or controllerthat interfaces with the processor 308. The touch screen controller 330may comprise a module configured to receive signals from a touch screenwhich is overlaid on the display 328.

Also part of this exemplary mobile device is a speaker 334 andmicrophone 338. The speaker 334 and microphone 338 may be controlled bythe processor 308. The microphone 338 is configured to receive andconvert audio signals to electrical signals based on processor 308control. Likewise, the processor 308 may activate the speaker 334 togenerate audio signals. These devices operate as is understood in theart and as such, are not described in detail herein.

Also connected to one or more of the buses 312 is a first wirelesstransceiver 340 and a second wireless transceiver 344, each of whichconnect to respective antennas 348, 352. The first and secondtransceivers 340, 344 are configured to receive incoming signals from aremote transmitter and perform analog front-end processing on thesignals to generate analog baseband signals. The incoming signal may befurther processed by conversion to a digital format, such as by ananalog to digital converter, for subsequent processing by the processor308. Likewise, the first and second transceivers 340, 344 are configuredto receive outgoing signals from the processor 308, or another componentof the mobile device 308, and up convert these signals from baseband toRF frequency for transmission over the respective antenna 348, 352.Although shown with a first wireless transceiver 340 and a secondwireless transceiver 344, it is contemplated that the mobile device 300may have only one or two such systems, or more transceivers. Forexample, some devices are tri-band or quad-band capable, or haveBluetooth®, NFC, or other communication capability.

It is contemplated that the mobile device 300, hence the first wirelesstransceiver 340 and a second wireless transceiver 344, may be configuredto operate according to any presently existing or future developedwireless standard including, but not limited to, Bluetooth, WI-FI suchas IEEE 802.11a,b,g,n, wireless LAN, WMAN, broadband fixed access,WiMAX, any cellular technology including CDMA, GSM, EDGE, 3G, 4G, 5G,TDMA, AMPS, FRS, GMRS, citizen band radio, VHF, AM, FM, and wirelessUSB.

Also part of the mobile device 300 is one or more systems connected tothe second bus 312B which also interfaces with the processor 308. Thesedevices include a global positioning system (GPS) module 360 withassociated antenna 362. The GPS module 360 is capable of receiving andprocessing signals from satellites or other transponders to generatedata regarding the location, direction of travel, and speed of the GPSmodule 360. GPS is generally understood in the art and hence notdescribed in detail herein. A gyroscope 364 connects to the bus 312B togenerate and provide orientation data regarding the orientation of themobile device 300. A magnetometer 368 is provided to supply directionalinformation to the mobile device 300. An accelerometer 372 connects tothe bus 312B to provide information or data regarding shocks or forcesexperienced by the mobile device. In one configuration, theaccelerometer 372 and gyroscope 364 generate and provide data to theprocessor 308 to indicate a movement path and orientation of the mobiledevice 300.

One or more cameras (still, video, or both) 376 are provided to captureimage data for storage in the memory 310 and/or for possibletransmission over a wireless or wired link, or for viewing at a latertime. The one or more cameras 376 may be configured to detect an imageusing visible light and/or near-infrared light. The cameras 376 may alsobe configured to utilize image intensification, active illumination, orthermal vision to obtain images in dark environments. The processor 308may process machine-readable code that is stored on the memory toperform the functions described herein.

A flasher and/or flashlight 380, such as an LED light, are provided andare processor controllable. The flasher or flashlight 380 may serve as astrobe or traditional flashlight. The flasher or flashlight 380 may alsobe configured to emit near-infrared light. A power management module 384interfaces with or monitors the battery 320 to manage power consumption,control battery charging, and provide supply voltages to the variousdevices which may require different power requirements.

FIG. 4 is a block diagram of an exemplary computing device, mobiledevice, or server, such as one of the devices described above, accordingto one exemplary embodiment. Computing device 400 is intended torepresent various forms of digital computers, such as smartphones,tablets, kiosks, laptops, desktops, workstations, personal digitalassistants, servers, blade servers, mainframes, and other appropriatecomputers. Computing device 400 is intended to represent various formsof mobile devices, such as personal digital assistants, cellulartelephones, smart phones, and other similar computing devices. Thecomponents shown, their connections and relationships, and theirfunctions, are meant to be exemplary only, and are not meant to limitthe implementations described and/or claimed in this document.

Computing device 400 includes a processor 402, memory 404, a storagedevice 406, a high-speed interface or controller 408 connecting tomemory 404 and high-speed expansion ports 410, and a low-speed interfaceor controller 412 connecting to low-speed bus 414 and storage device406. Each of the components 402, 404, 406, 408, 410, and 412, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 402 canprocess instructions for execution within the computing device 400,including instructions stored in the memory 404 or on the storage device406, to display graphical information for a GUI on an externalinput/output device, such as display 416 coupled to high-speedcontroller 408. In other implementations, multiple processors and/ormultiple buses may be used, as appropriate, along with multiple memoriesand types of memory. Also, multiple computing devices 400 may beconnected, with each device providing portions of the necessaryoperations (e.g., a server bank, a group of blade servers, or amulti-processor system).

The memory 404 stores information within the computing device 400. Inone implementation, the memory 404 is a volatile memory unit or units.In another implementation, the memory 404 is a non-volatile memory unitor units. The memory 404 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 406 is capable of providing mass storage for thecomputing device 400. In one implementation, the storage device 406 maybe or contain a computer-readable medium, such as a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid-state memory device, or an array of devices, including devices ina storage area network or other configurations. A computer programproduct can be tangibly embodied in an information carrier. The computerprogram product may also contain instructions that, when executed,perform one or more methods, such as those described above. Theinformation carrier is a computer- or machine-readable medium, such asthe memory 404, the storage device 406, or memory on processor 402.

The high-speed controller 408 manages bandwidth-intensive operations forthe computing device 400, while the low-speed controller 412 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In one implementation, the high-speed controller 408 iscoupled to memory 404, display 416 (i.e., through a graphics processoror accelerator), and to high-speed expansion ports 410, which may acceptvarious expansion cards (not shown). In this representativeimplementation, low-speed controller 412 is coupled to storage device406 and low-speed bus 414. The low-speed bus 414, which may includevarious communication ports (e.g., USB, Bluetooth, Ethernet, wirelessEthernet) may be coupled to one or more input/output devices, such as akeyboard, a pointing device, a scanner, or a networking device such as aswitch or router (i.e., through a network adapter).

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 420, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 424. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 422. Alternatively, components from computing device 400 may becombined with other components in a mobile device (not shown), such asdevice 450. Each of such devices may contain one or more computingdevices 400, 450, and an entire system may be made up of multiplecomputing devices 400, 450 communicating with each other.

Computing device 450 includes a processor 452, memory 464, aninput/output device such as a display 454, a communication interface466, and a transceiver 468, among other components. The computing device450 may also be provided with a storage device, such as a micro-drive orother device(s), to provide additional storage. Each of the components452, 464, 454, 466, and 468, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 452 can execute instructions within the computing device450, including instructions stored in the memory 464. The processor 452may be implemented as a chipset of chips that include separate andmultiple analog and digital processors. The processor may provide, forexample, for coordination of the other components of the computingdevice 450, such as control of user interfaces, applications run by thecomputing device 450, and wireless communication by the computing device450.

Processor 452 may communicate with a user through control interface 458and display interface 456 coupled to a display 454. For example, thedisplay 454 may be a TFT LCD (Thin-Film-Transistor Liquid CrystalDisplay) or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. The display interface 456 may compriseappropriate circuitry for driving the display 454 to present graphicaland other information to a user. The control interface 458 may receivecommands from a user and convert them for submission to the processor452. In addition, an external interface 462 may be provided incommunication with processor 452, to enable near area communication ofcomputing device 450 with other devices. In some implementationsexternal interface 462 may provide for wired communication, or in otherimplementations, for wireless communication, whilst multiple interfacesmay also be used.

The memory 464 stores information within the computing device 450. Thememory 464 can be implemented as one or more of a computer-readablemedium or media, a volatile or a non-volatile memory unit or units.Expansion memory 474 may also be provided and connected to the computingdevice 450 through expansion interface 472, which may include, forexample, a SIMM (Single In Line Memory Module) card interface. Suchexpansion memory 474 may provide extra storage space and/or may alsostore applications or other information for the computing device 450.Specifically, expansion memory 474 may include instructions to carry outor supplement the processes described above and may also include secureinformation. Thus, for example, expansion memory 474 may be provided asa security module for computing device 450 and may be programmed withinstructions that permit secure use of the same. In addition, secureapplications may be provided via the DIMM cards, along with additionalinformation, such as placing identifying information on the DIMM card ina non-hackable manner.

The memory may include for example, flash memory and/or NVRAM memory, asdiscussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 464, expansionmemory 474, or memory on processor 452, that may be received forexample, over transceiver 468 or external interface 462.

The computing device 450 may communicate wirelessly throughcommunication interface 466, which may include digital signal processingcircuitry where necessary. Communication interface 466 may provide forcommunications under various modes or protocols, such as GSM voicecalls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, orGPRS, among others. Such communication may occur for example, through aradio-frequency transceiver 468. In addition, short-range communicationmay occur, such as using a Bluetooth, Wi-Fi, or other such transceiver(not shown). In addition, GPS (Global Positioning system) receivermodule 470 may provide additional navigation- and location-relatedwireless data to the computing device 450, which may be used asappropriate by applications running on the computing device 450.

The computing device 450 may also communicate audibly using audio codec460, which may receive spoken information from a user and convert it tousable digital information. Audio codec 460 may likewise generateaudible sound for a user, such as through a speaker, e.g., in a handsetof the computing device 450. Such sound may include audio from voicetelephone calls, recorded audio (e.g., voice messages, music files,etc.), and may also further include audio generated by applicationsoperating on the computing device 450.

The computing device 450 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 460. It may also be implemented as part of asmartphone 482, personal digital assistant, a computer tablet, or othersimilar mobile device.

Thus, various implementations of the systems and techniques describedhere can be realized in digital electronic circuitry, integratedcircuitry, especially designed ASICs (application specific integratedcircuits), computer hardware, firmware, software, and/or combinationsthereof. These various implementations can include applications in oneor more computer programs that are executable and/or interpretable on aprogrammable system, including at least one programmable processor whichmay be special or of general purpose, coupled to receive data andinstructions, to and from a storage system, at least one input device,and at least one output device.

These computer programs (also known as programs software, softwareapplications or code) include machine instructions for a programmableprocessor and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display deviceLCD (liquid crystal display) monitor, LED, or any other flat paneldisplay, for displaying information to the user, a keyboard, and apointing device (e.g., mouse, joystick, trackball, or similar device) bywhich the user can provide input to the computer. Other kinds of devicescan be used to provide for interaction with a user as well, for example;feedback provided to the user can be any form of sensory feedback (e.g.,visual, auditory, or tactile); and input from the user can be receivedin any form, including acoustic, speech, or tactile input.

The systems and techniques described here may be implemented in acomputing system (e.g., computing device 400 and/or 450) that includes aback end component, or that includes a middleware component (e.g.,application server), or that includes a front-end component such as aclient computer having a graphical user interface or a Web browserthrough which a user may interact with an implementation of the systemsand techniques described herein, or any combination of such back-end,middleware, or front-end components. The components of the system may beinterconnected by any form or medium of digital data communication, suchas a communication network. Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

It will be appreciated that the virtual personification AI system may beused to implement many possible applications. For example, aconventional weather application may display weather searchable bypre-determined parameters (such as location and date). In contrast, theAI system may output a virtual personification of a popular on-camerameteorologist (such as Jim Cantore) to not only provide weather based onpre-determined parameters, but also to further interact with the user.For example, a user may first request the current weather and a forecastfor Hawaii, and then ask, “what clothes should I pack for my upcomingvacation”? A conventional weather application will not understand theuser question because it may not remember the context (weather inHawaii). The virtual personification AI system, on the other hand, maynot only understand the context of the user question and accuratelydetermine the user's true request—a retrieval of the user's calendar todetermine the accurate date range and/or location for the upcomingvacation, a projection of the weather during that date range, ananalysis of proper attire given the weather, personal data (such as userpreferences on clothing items), and location (which may take intoaccount additional factors such as humidity, altitude, and localculture). Further, the AI may be capable of presenting the properresponse to the user request using the virtual personification of JimCantore in a conversational format, even though that person has notpreviously answered that question or provided that particular responsein the past.

It is contemplated that pre-recorded data may be analyzed, and virtualpersonifications may be generated, for any person(s), not just famousones. For example, a user may submit family videos of agreat-grandfather who has passed away, and the virtual personificationsystem may render a virtual personification of the great-grandfather,who may interact with future generations. This concept may be applied toany person living or passed away. In addition, the system may createvirtual personifications which have an appearance different to anyonealive or previously alive.

In one embodiment, the AI may supplement missing footage with its own.In the example where a user asks the virtual personification of JillianMichaels, a fitness coach, for a standing core exercise, but norecording of Jillian Michaels performing such an exercise exists, thennew rendering may be generated using Jillian Michaels' footageperforming other, but similar actions, or other people performing therequested exercise, but rendered to appear as Jillian Michaels. The newrendering may be generated using deepfake or other technology to combineJillian Michaels' footage with generic footage of another personperforming a standing core exercise. As yet another alternative, the AImay provide a set of default footage (such as a default virtual modelperforming standing core exercises) and superimpose whatever footage ofJillian Michaels' may be available on the default model. It iscontemplated that any other type of technology may be used to generatenew rendering where no pre-recorded data exist.

Another possible expansion of the virtual personification AI system maybe to generate and render entirely imaginary characters. For example, bycombining a new character design for a two-headed animal with defaultmodels or existing footage of a dinosaur, a virtual personification of anew two-headed dinosaur may be generated, and its behavior may be basedon analysis of a wolf, or it may interact with the user with the voiceof an old man. Thus, virtual personification may be customized based onboth pre-recorded data and user preferences. It is contemplated that thepersonification is not limited to people but other things (real or notreal), such as but not limited to animals, robots, cars, birds, fish,extinct species, alien creatures, created items or beings, or any otheritem.

The virtual personification AI system's ability to generate any type ofvirtual personification and its versatility of customization enablesbroad application to all types of technology and environment. Oneexemplary use may be education, where classes and/or tutorials may beprovided to students with virtual personification of an instructor onany subject, which may automate some or all of a student's classroomexperience without compromising the student's personal interaction (suchas to ask questions). As an improvement to in-person experiences, thevirtual instructor may draw information from existing knowledge ordatabases, thereby providing answers a live instructor may not have. Theclass may be taught by a virtual representation of someone famous, suchas Albert Einstein, Sandra Day O'Connor, or Alexander Graham Bell.

Another exemplary use may be training, which may include standardizedtraining for professional purposes (such as customized professionaltraining of airplane pilots using a virtual personification of a flightinstructor and the cockpit), or training for hobbies or informationlearning (such as cooking with a virtual personification of GordonRamsey). In the case of an airplane, the virtual environment may reactto different actions by the user, such as use of certain controls, toprovide realistic training.

Similarly, the virtual personification AI system may be used to generatereal-time response instructions, such as medical or emergency training.For example, a 9-1-1 dispatcher may assist a caller to perform CPR bytransmitting footage of a virtual personification of medical personnelperforming CPR on a patient while waiting for an ambulance to arrive.The caller may interact with the medical personnel by asking questionssuch as “the patient is still not breathing, now what”? Answers may bepulled from a database and the personification may actually perform theanswer to the question so the user can see how to perform the medicaltraining. This concept can be applied to teaching tools for medicalprocedures such that the virtual representations of the best doctors inthe world can be created to show how to perform a procedure anddynamically respond to any question for personalized teaching.

Yet another exemplary use may be entertainment, such as allowing usersto interact with famous people from the past or an imaginary characterin a conversational setting. This allows people to have personalinteractions with people from history and interact using the AIdatabases such that the virtual representation can answer any questionor perform any action in real time during user interaction. These famouspeople from which the virtual representation is created could be anyperson, famous for any reason. A database may be created for each personso that the AI system can accurately create visual, audio, and knowledgerepresentations.

Similarly, the virtual personification AI system may be used to generatereal-time response instructions, such as medical or emergency training.For example, a 9-1-1 dispatcher may assist a caller to perform CPR bytransmitting footage of a virtual personification of medical personnelperforming CPR on a patient while waiting for an ambulance to arrive.The caller may interact with the medical personnel by asking questionssuch as “the patient is still not breathing, now what”?

In future expansions of the technology, it may also be possible tostimulate virtual personification without user input. For example, anentire lecture series may be generated using virtual personification ofan instructor and combining footage of lecture recordings of pastreal-life lectures. Anticipatory answers may be provided by analysis ofrecording of past student questions, thereby eliminating the need forfurther student interaction (which may still be provided as anadditional feature).

Yet another example may be a simulated room where two virtualpersonifications may interact with each other instead of (or in additionto) interaction with users. For example, a user may wish to simulate aphilosophical debate between Socrates and Kant. In one example, thevirtual room may be expanded to include entire virtual worlds and largepopulations of virtual characters. The user can learn from seeing howtwo or more virtual representation interact, such as in professionalenvironments, military situation, formal engagements, or socialinteraction.

While various embodiments of the invention have been described, it willbe apparent to those of ordinary skill in the art that many moreembodiments and implementations are possible that are within the scopeof this invention. In addition, the various features, elements, andembodiments described herein may be claimed or combined in anycombination or arrangement.

What is claimed is:
 1. A method to generate and update virtualpersonification using artificial intelligence comprising the steps of:receiving data associated with a person, the data comprising one or moreof the following: text files, audio files, image files, and video files;rendering a virtual personification of the person and outputting thevirtual personification to a user; receiving and interpreting a userinput to generate a user request; updating the virtual personificationin response to the user request, the update comprising one or more ofthe following: responsive to the user request, generating an audiooutput using the text files and the audio files of the person; andresponsive to the user request, generating a video output using theimage files and the video files of the person, wherein the audio outputand the video output is presented to the user by the virtualpersonification and the audio output and the video output presented bythe virtual personification has not previously occurred by the person orthing represented by the virtual personification.
 2. The method of claim1 wherein the virtual personification is of a person, either living ordeceased.
 3. The method of claim 1 wherein the virtual personificationcomprises an audio output and video output is presented in a virtualenvironment of a type associated with the virtual personification. 4.The method of claim 1 wherein the virtual personification comprises arepresentation of a non-living item.
 5. The method of claim 1 whereinthe method is further configured to, responsive to being unable tocreate the generated response at the virtual reality device, transmitthe question or request from the user to a remote artificialintelligence module.
 6. The method of claim 5, wherein the remoteartificial intelligence module is a computing device with a processorand memory storing machine readable code configured to: receive thequestion or request from the user via the virtual reality device;process the question or request to derive a meaning; perform one or moresearches for answers to the question or request in databases unrelatedto the virtual personification; upon locating an answer to the questionor request, generating data that represents the virtual personificationanswering the question or request; and transmitting the answer or thedata that represents the virtual personification answering the questionor request to the virtual reality device for presentation to the user.7. The method of claim 1 further comprising tracking a hand position ofa user with one or more user hand position tracking devices to determinewhat the user is pointing at in the virtual environment.
 8. The methodof claim 1 wherein the generated response to the question or requestuses artificial intelligence to generate an answer by searching one ormore databases that contain information from the person represented bythe virtual personification but which does not provide a direct answerto the question or request.
 9. A system for presenting an interactive,artificial intelligence assisted, virtual personification to a usercomprising:
 10. A virtual reality device configured to have at least aportion be worn by the user comprising: a wearable screen configured forviewing by a user; one or more speakers configured to provide audiooutput to the user; a microphone configured to receive audio input fromthe user; one or more external input devices configured to receive inputfrom the user; a communication module configured to configured tocommunicate over a computer network or Internet; a processor configuredto execute machine readable code; a memory configured to store themachine readable code, the machine readable code configured to: presenta virtual environment on the wearable screen and through the one or morespeakers to the user; present, to the user on the wearable screen andthrough the one or more speakers, a virtual personification of a personcurrently living or deceased, in the virtual environment; receive aquestion or request from the user regarding one or more aspects of thevirtual environment or the virtual personification; generate a generatedresponse to the question or request from the user which includesgenerating video content and audio content which did not previouslyexist; present the generated response to the user on the wearable screenand through the one or more speakers in response to question or requestfrom the user.
 10. The system of claim 9 wherein the machine readablecode is further configured to, responsive to being unable to create thegenerated response at the virtual reality device, transmit the questionor request from the user to a remote artificial intelligence module. 11.The system of claim 10 wherein the remote artificial intelligence moduleis a computing device with memory and processor such that the memorystores machine readable code configured to: receive the question orrequest from the user via the virtual reality device; process thequestion or request to derive a meaning; perform one or more searchesfor answers to the question or request in databases unrelated to thevirtual personification; upon locating an answer to the question orrequest, generate data that represents the virtual personificationanswering the question or request; and transmit the answer or the datathat represents the virtual personification answering the question orrequest to the virtual reality device for presentation to the user. 12.The system of claim 9 further comprising one or more user hand positiontracking devices configured to track a position of a user's hand todetermine what the user is pointing at in the virtual environment. 13.The system of claim 9 wherein the input from the user comprises an audioinput or an input from the user to the one or more external inputdevices.
 14. The system of claim 9 wherein generating video content andaudio content which did not previously exist is generated by processingexisting video, audio, or both of the person represented by the virtualpersonification.
 15. The system of claim 9 wherein the generatedresponse to the question or request is generated using artificialintelligence to generate an answer by searching one or more databasesthat contain information from the person represented by the virtualpersonification but which does not provide a direct answer to thequestion or request.
 16. A method for presenting an interactiveexperience with a virtual personification using a screen, speakers, andmicrophone of a user computing device, the method comprising: presentinga virtual environment on the wearable screen and through the one or morespeakers to the user and presenting the virtual personification in thevirtual environment; receiving input from the user comprising aquestion, a request, or subject regarding one or more aspects of thevirtual environment, the virtual personification, or the actions of thevirtual personification in the virtual environment; sending a requestfor a response to the input from the user to an AI computing device thatis remote from the user computing device; with the AI computing device,creating a response based on pre-existing content stored in one ordatabases which is processed to create the generated response;transmitting the generated response to the user computing device; at theuser computing device, based on the generated response from the AIcomputing device, generating video content and audio content which didnot previously exist; and presenting video content and audio contentwhich did not previously exist to the user.
 17. The system of claim 16wherein the AI computing device is a computing device with memory andprocessor such that the memory stores machine readable code configuredto: receiving the input from the user computing device; processing theinput from the user to derive a meaning; based on the meaning,performing one or more searches for answers to the input from the userin databases unrelated to the virtual personification; upon locating aresponse to the input from the user, generating data that represents thevirtual personification answering the question or request; andtransmitting the data, that represents the virtual personificationresponding to the input from the user to the user computing device. 18.The system of claim 16 further monitoring one or more user hand positiontracking devices configured to track a position of a user's hand todetermine what the user is pointing at in the virtual environment andinterpreting the pointing as the input from the user.
 19. The system ofclaim 16 wherein the input from the user comprises an audio input or aninput from the user to the one or more external input devices.
 20. Thesystem of claim 16 wherein generating video content and audio contentwhich did not previously exist is generated by processing existingvideo, audio, or both of a person represented by the virtualpersonification to generate new content.